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Overview 


■  Background 

■  Understanding  the  Value  Added  Analysis  Process 

■  Challenges 

■  Demand  for  quick-turnaround  analysis 

■  Non-monotonic  relationships  between  parameters  and  VIC 
outcomes 

■  Statistical  analysis  to  support  accurate  decision  making 

■  Our  Answer 

■  Induce  stochastic  variation  in  VIC  runs  through  perturbation 

■  Utilize  statistical  tests  for  comparisons  of  options 

■  Benefit 

■  More  accurate  decisions  about  equipment  trades  using  VIC 
as  part  of  the  Value-Added  Analysis  process 
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The  Value  Added  Analysis  Process 


Supporting  the  Center  for 
Army  Analysis  (CAA) 

■  Uses  the  Vector-in- 
Commander  (VIC)  Corps- 
level  combat  simulation 
model 

■  Objective: 

■  Estimate  the  incremental 
contribution  of  system 
trades  to  combat 
effectiveness 

■  Perform  a  cost-benefit 
analysis  to  determine  the 
actual  'value-added’  of 
the  systems  of  interest. 


Value  Added  Analysis 


Previous  methodology  was  a  R9urc  1-  ^  VAA  P^cess 
typical  DOE  approach 


Now  a  perturbation 
methodology  induces 
stochastic  Behavior  in  VIC 
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The  Force  Exchange  Ratio 


■  Primary  Measure  of  Effectiveness 
(MOE)  in  the  VAA  process 

■  Force  Exchange  Ratio  (FER) 

■  Ratio  of  relative  losses 


FER  = 


LossesR/ 

/ Strength  R 

LossesB/ 

/ Strength  B 


■  Used  as  a  proxy  for  the  win 
probability 

■  Only  high-value  equipment 
losses  are  counted  in  our 
version 
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Legacy  Methodology 


Comparing  FERs  using  a 
Design  of  Experiments 

A  typical  DOE  is  to  run  a 
number  of  combinations  of 
experimental  settings 

■  And  then  analyze  the  MOEs 
using  analysis  of  variance 

■  Differences  in  the  means 
between  treatments  indicate 
possible  differences  in 
effectiveness 
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DOEs  aid  in  making  statistical 
“decisions”: 


Statistical  efficiency  is 
achieved  at  the  cost  of 
elaborate  run  setups. 


FER(X.|  =  1)  -  FER(X1  =  -1)  >  0  ? 


How  big  is  the  difference? 

What  is  the  confidence  interval? 
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Two  Paradigm  Shifts 


■  VIC  and  complexity 

■  Battle  is  a  complex  dynamical  system 

■  The  results  of  battle  are  somewhat  uncertain 

■  Especially  when  the  foes  are  close  to  evenly  matched 

■  VIC  battles  are  a  realization  of  a  complex  dynamical 
system 

■  Sensitivity  to  parameters  and  initial  conditions 
should  be  expected 


■  Embrace  complexity 

■  In  support  of  quick  turnaround  analysis 

■  Using  the  statistical  perspective 
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Inducing  the  Expected  Variability 


■  Statistical  methods  require  variability  and 
replication. 

■  Key  Requirements  for  inducing  stochastic 
behavior  consistent  with  accurate  analysis: 

■  Must  not  alter  any  performance  data  (Bailey,  2001) 

■  Must  affect  many  battlefield  operating  systems 
(Bailey,  2001) 

■  Must  continuously  perturb  the  run  -  not  just  the 
initial  conditions  (Bailey,  2001) 

■  Retains  the  original  scenario  setup  within  the 
precision  of  combat  operations 
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A  Perturbation  Methodology 


■  Our  method  perturbs  several  things  a  “small” 
amount 

■  Unit  locations  and  waypoints 

■  Helo  path  points 

■  Airborne  sensor  orbit  points 

■  See  Bulanow  et  al.  (2004)  for  validation  with 
respect  to  using  the  outputs  in  statistical 
models 
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The  Difficulty  with  the  Two-Run  Comparison 


Non-Monotonic  effects  have  been 
observed  in  Deterministic  Combat 
Models 

■  Better  settings  do  not 
necessarily  mean  a  better  FER 

■  Sensitivity  to  initial  conditions  and 
parameter  values 

■  Extensively  noted  in  toy  models 
of  combat 

■  The  RAND  model  (Dewar,  et 
al,  1991) 

■  Also  noted  in  VIC 


Kills  of  Selected  Equipment  by  a 
Blue  System  of  Interest 


■  Saeger  &  Hinch  (2001) 

■  Geoff  Hawkins  (1984)  with 
VECTOR-2 


A  three-way  comparison 
of  VIC  results 


The  DOE  is  a  legacy  solution  to  this 
problem,  but  a  more  responsive 
approach  is  required. 
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Parametric  Sensitivity  in  VIC 


A  direct  fire  system 
(DF  Sys)  fraction  of 
time  firing  (FTF)  is 
multiplied  by  a 
number  randomly 
selected  from  the 
interval  (0.95, 1.05) 

■  Blue  kills  vary  non- 
monotonically  and 
significantly 

■  Any  two  runs 
selected  from  these 
might  show  a 
difference  in  the  MOE 

■  But  is  the 
difference 
statistically 
“significant”? 


Kills  by  Blue  of  High-Value  Equipment 
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Direct  Fire  System  Fraction  of  Time  Firing  Multiplier 
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Inducing  Variability  Through  Unit  Locations 


Kills  by  Blue  and  Red  of  High-Value  Equipment 


Perturbing  ground  unit 
locations  and  waypoints 
by  ±10  meters  produces 
very  different  pictures  of 
the  loss  exchange  ratio. 

Each  color  line 
represents  the  plot  of 
Blue  versus  Red  kills 
over  the  run  for  the 
original  and  64 
replications 

■  X  and  Y  scales 
include  zero  but  are 
not  the  same 
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More  on  VIC’s  Stochastic  Personality 


Perturbing  ground 
unit  locations 
small  amounts  (a 
non-performance 
parameter)  reveals 
a  world  of 
stochastic 
variability 

■  Like  what 
might  happen 
in  combat 


Statistical 
methods  can 
characterize  this 
variability  for 
decision-making 
purposes 


Kills  of  Selected  Equipment  by  the  Blue  System  of  Interest 


Base  Case,  Perturbed 


Hi  i  Nimber 
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Analysis  Without  DOE  Matrices 


■  Paired  Comparisons  can  be  performed  without  an 
elaborate  DOE 


Perturbation 
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■  We  also  perform  multiple  comparisons  between 
numerous  options 

■  More  efficient  for  the  analyst  due  to  fewer  run 
setups  than  with  a  DOE 

■  Has  been  employed  in  a  variety  of  trade 
comparisons 
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Effect  of  Replicat  ons  on  the  Confidence 
Interval  of  Estimates 


■  Confidence 
intervals 
decrease  as  the 
inverse  square 
root  of  sample 
size 

■  In  actual 
applications,  the 
standard 
deviation  would 
be  estimated 


#  Replications 
(and  run  time 
factor) 

Confidence  Interval 
(assuming  a  notional 
standard  deviation, 
known  in  advance) 

1 

±32% 

4 

±16% 

16 

±8% 

64 

±4% 

256 

±2% 
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Conclusions  and  Way  Forward 

■  Our  perturbation  analysis  for  VIC 
analysis  aids  in  quick-turn  analysis  by: 

■  Reducing  run  setups, 

■  Simplifying  design  and  analysis  of 
experiments,  ana 

■  Enabling  statistical  analysis  with 
simple  designs 

■  VIC  run  perturbation  gives  visibility  to 
the  complex  system  feature  of  combat 

■  Even  though  VIC  is  deterministic 

■  Thus  providing  an  added  window 
into  the  issue  of  outcome  variability 
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